Scene Text Recognition

Posted on | In

Scene text detection and recognition are challenging due to the following issues: scattered and sparse, blur, illumination, partial occlusion, multi-oriented, multi-lingual.

Scene text detection:

The detection methods can be grouped into proposal-based method and part-based method.

Paper list (in chronological order):

  1. Detecting Text in Natural Scenes with
    Stroke Width Transform    , CVPR 2010: assume consistent stroke width within each character

  2. Detecting Texts of Arbitrary Orientations in Natural Images, CVPR 2012: design rotation-invariant features

  3. Deep Features for Text Spotting, ECCV 2014: add three branches for prediction

  4. Robust scene text detection with convolution neural network induced mser trees, ECCV 2014

  5. Real-time Lexicon-free Scene Text
    Localization and Recognition    , T-PAMI 2016

  6. Reading Text in the Wild with Convolutional Neural Networks, IJCV 2016

  7. Synthetic Data for Text Localisation in Natural Images, CVPR 2016: directly predict the bounding boxes, generate synthetic dataset

  8. Multi-oriented text detection with fully convolutional networks, CVPR 2016

  9. Detecting Text in Natural Image with Connectionist Text Proposal Network, ECCV 2016: look for text lines an fine vertical text pieces. sliding windows fed to Bi-LSTM.

  10. SSD: single shot multibox detector, ECCV 2016

  11. Reading Scene Text in Deep Convolutional Sequences, AAAI 2016

  12. Scene text detection via holistic, multi-channel prediction, arxiv 2016: holistic and pixel-wise predictions on text region map, character map, and linking
    orientation map

  13. Deep Direct Regression for Multi-Oriented Scene Text Detection, ICCV 2017

  14. WordSup: Exploiting Word Annotations for Character based Text Detection, ICCV 2017: a weakly supervised framework that can utilize word annotations for character detector training

  15. TextBoxes: A Fast Text Detector with a Single Deep Neural Network, AAAI 2017

  16. Detecting Oriented Text in Natural Images by Linking Segments, CVPR 2017: detect text with segments and links

  17. EAST: An Efficient and Accurate Scene Text Detector, CVPR 2017: use DenseBox to generate quadrangle proposals

  18. TextBoxes++: A Single-Shot Oriented Scene Text Detector, TIP 2018: extension of TextBoxes

  19. Rotation-sensitive Regression for Oriented Scene Text Detection, CVPR 2018: rotation-sensitive feature maps for regression and rotation-invariant features for classification

  20. Multi-Oriented Scene Text Detection via Corner Localization and Region Segmentation, CVPR 2018: combine corner localization and region segmentation

  21. PixelLink: Detecting Scene Text via Instance Segmentation, AAAI 2018: rectangle enclosing instance segmentation mask, which is obtained based on text/non-text prediction and link prediction.

  22. Arbitrary-Oriented Scene Text Detection via Rotation Proposals, TMM 2018: generate rotated proposals

  23. TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapes, ECCV 2018: infer the center line area (TCL) and associated circle radius/rotation

Scene text recognition:

The recognition methods can be grouped into character-level, word-level, and sequence-level.

Paper list (in chronological order):

  1. End-to-End Scene Text Recognition, ICCV 2011: detection using Random Ferns and recognition via Pictorial Structure with a Lexicon

  2. Top-down and bottom-up cues for scene text recognition, CVPR 2012: construct a CRF model to impose both bottom-up (i.e. character detections) and top-down (i.e. language statistics) cues

  3. Scene text recognition using part-based tree-structured character detection, CVPR 2013: build a CRF model to incorporate the detection scores, spatial constraints and linguistic knowledge into one framework

  4. PhotoOCR: Reading text in uncontrolled conditions, ICCV 2013: automatically generate training data and perform OCR on web images

  5. Label embedding: A frugal baseline for text recognition, IJCV 2015: learn a common space for image and word

  6. Reading Text in the Wild with Convolutional Neural Networks, IJCV 2016

  7. Robust Scene Text Recognition with Automatic Rectification, CVPR 2016

  8. Recursive Recurrent Nets with Attention Modeling for OCR in the Wild, CVPR 2016: character-level language model embodied in a recurrent neural network

  9. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition, T-PAMI 2017

  10. Focusing Attention: Towards Accurate Text Recognition in Natural Images, ICCV 2017: Focusing Network to handle the attention drift

  11. Visual attention models for scene text recognition, 2017 arxiv

  12. AON: Towards Arbitrarily-Oriented Text Recognition , CVPR 2018

  13. (recommended by Guo)An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition, T-PAMI, 2017

End-to-end

Integrate scene text detection and recognition in an end-to-end system.

Paper list (in chronological order):

  1. A method for text localization and recognition in real-world images, ACCV 2010

  2. Real-Time Scene Text Localization and Recognition, CVPR 2012

  3. Towards End-to-end Text Spotting with Convolutional
    Recurrent Neural Networks    , ICCV 2017: designed for horizontal scene text

  4. Deep TextSpotter: An End-To-End Trainable Scene Text Localization and Recognition Framework, ICCV 2017: detect and recognize horizontal and multioriented
    scene text

  5. FOTS: Fast Oriented Text Spotting with a Unified Network, CVPR 2018: using EAST as text detector and CRNN as text recognizer

Datasets

Surveys

Special Sessions

  1. Use Spatial Transformation Network (STN) [1] [2] [3] [4]

  2. Use Deformable Convolution Network (DCN) [1]